A Probabilistic Approach to the Semantic Interpretation of Building Facades

Semantically-enhanced 3D model reconstruction in urban environments is useful in a variety of applications, such as extracting metric and semantic information about buildings, visualizing the data in a way that outlines important aspects, or urban planning. We present a probabilistic image-based approach to the semantic interpretation of building facades. We are motivated by the 4D Atlanta project at Georgia Tech, which aims to create a system that takes a collection of historical imagery of a city and infers a 3D model parameterized by time. Here it is necessary to recover, from historical imagery, metric and semantic information about buildings that might no longer exist or have undergone extensive change. Current approaches to automated 3D model reconstruction typically recover only geometry, and a systematic approach that allows hierarchical classification of structural elements is still largely missing. We extract metric and semantic information from images of facades, allowing us to decode the structural elements in them and their inter-relationships, thus providing access to highly structured descriptions of buildings. Our method is based on constructing a Bayesian generative model from stochastic context-free grammars that encode knowledge about facades. This model combines low-level segmentation and high-level hierarchical labelling so that the levels reinforce each other and produce a detailed hierarchical partition of the depicted facade into structural blocks. Markov chain Monte Carlo sampling is used to approximate the posterior over partitions given an image. We show results on a variety of real images of building facades. While we have currently tested only limited models of facades, we believe that our framework can be applied to much more general models, and are currently working towards that goal.